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PREFACE 


In  February,  1980,  the  Evaluation  Unit   within  the 
Policy  and  Planning  organization  line  was  discontinued  as 
a  distinct  entity  and  the  responsibilities  were  assigned 
to  the  Research  Division. 

In  March,  1980,  Harold  Horowitz,  Director,  Research 
Division,  inquired  about  my  interest  in  working  with  him 
during  the  planning  period  as  he  developed  a  systematic 
evaluation  study  program.   Since  evaluation  study  approaches 
are  a  long-term  interest  of  mine  in  work  along  this  line 
for  the  U.S.  Office  of  Education  and  earlier,  the  National 
Science  Foundation,  I  enthusiastically  accepted.   We  agreed 
that  in  25  days  I  would  review  completed  and  on-going 
evaluation  studies  and  make  recommendations  for  improvement. 

I  have  read  stacks  of  reports,  documents  and  guides. 
Possibly,  I  became  involved  in  substance  and  content  in 
greater  depth  than  necessary.   Program  directors  and  other 
NEA  officials  freely  discussed  evaluation  problems  with  me, 
and  I'm  sure  when  they  saw  my  interest,  they  willingly 
extended  their  discussions  to  include  the  excitment  of  their 
activities.   Also,  I  have  held  discussions  with  individuals 
outside  the  NEA  staff  who  are  knowledgeable  about  evaluation 
in  the  arts.   Mainly,  in  this  project  I  have  sought  ideas 


from  others,  talked  about  my  experiences  and  in  this  report 
have  included  my  biases.   The  Research  staff  have  generously 
given  me  as  much  time  as  I  needed.   Bill  Potter's  help  has 
been  invaluable,  as  has  been  the  assistance  of  Candice 
Parrish  and  Maryann  Gerard.   Thanks  to  everybody  concerned. 

Early  in  my  work  I  became  convinced  that  the  assignment 
could  be  expanded  and  that  more  would  be  gained  than  by 
concentrating  all  of  my  time  and  attention  on  completed  and 
on-going  studies.   I  discussed  with  Mr.  Horowitz  my  desire 
to  range  farther  and  consider  some  factors  in  launching  a 
viable  system  of  evaluation  studies.   Section  1  covers  that 
aspect  of  my  thinking.   The  purpose  is  to  state  the  many  areas 
that  should  be  included  in  a  comprehensive  evaluation  program. 
The  details  for  the  parts  are  not  presented.   Consideration 
has  not  been  given  to  scaling  the  overall  plan  to  mesh  with 
resources  likely  to  be  available.   Section  2  deals  with 
on-going  and  completed  studies.   The  discussions  that  I  have 
held  with  Harold  and  will  hold  in  the  near  future  will  be  a 
main  contribution.   This  report  is  directed  to  him  for  his 
consideration . 


Denzel  D.  Smith 
Senior  Consultant 
University  Associates,  Inc. 
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Section  1 


The  Setting 

It  is  not  the  purpose  of  this  report  to  reinvent  the 
term  evaluation,  enumerate  technical  strategies,  or  debate 
the  effectiveness  of  particular  evaluation  methodologies. 
Evaluation  of  some  kind  happens,  as  it  always  has,  for  any 
program.   We  attach  different  values  to  the  ranges  of 
information  available  for  decision-making  situations.   Some 
leaders  feel  that  the  realities  of  political  imperatives 
deny  the  uses  of  impact  evaluation,  for  example.   However, 
the  timing  seems  right  to  seek  an  improved  vantage  point 
for  understanding  the  extent  to  which  the  Endowment  is 
meeting  stated  goals. 

Similar  statements  are  expressed  regularly  concerning 
the  planning  function.   Yet,  the  commitment  to  effective 
planning  for  the  Endowment  has  been  made.   To  many  people 
neatness  of  the  exercise  in  planning  is  not  considered  to  be 
very  important  in  the  decision  making  process.   To  them  the 
exercise  simply  provides  better  copy  for  presentations,  for 
example,  budget  presentations.   Also,  precisely  how  information 
on  program  effectiveness  and  impact  can  be  used  to  facilitate 
management  decisions  is  not  well  communicated  and  often  not 
believed  when  communicated. 
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The  Endowment  is  committed  to  extensive  planning  —  short 
range,  middle  range  and  long  range.   I  have  discerned,  however, 
that  it  appears  less  committed  to  development  of  evaluation 
in  a  systematic  manner  as  a  means  of  better  understanding 
progress  toward  achieving  goals. 

Structure  and  Substance 

What,  then,  should  be  the  structure  and  substance  of  an 
organizational  unit  with  responsibility  for  providing 
important  evaluation  information.' 

First,  all  levels  within  an  agency  should  accept 
responsibilities  for  demonstrating  progress  toward  meeting 
their  objectives  and  the  agency's  goals.   This  is  a  coopera- 
tive venture.   As  program  planning  begins,  attention  should 
be  given  to  stating  objectives;  the  main  strategies  should 
be  named  and  some  indicators  identified  that  show  progress. 
A  key  factor  is  that  evaluation  requirements  should  be 
identified  during  early  planning  phases. 

The  evaluation  staff  of  the  Research  Division  should  have 
the  competence  to  provide  technical  assistance  to  program  leaders 
in  the  form  of  training  when  requested  and  advice  on  how  achieve- 
ment might  be  considered.   The  function  should  be  cast  at  the 
program  level  in  terms  of  facilitating  effectiveness  of  a 
program;  and  at  the  agency  level  facilitating  support  toward 
goal  attainment.   The  supportive  service  would  not  duplicate 
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other  services  or  assume  responsibilities  of  other  offices 
in  preparing  program  objectives  or  identifying  measures  of 
progress . 

The  Planning  and  Budget  Division,  of  course,  has  primary 
responsibilities  in  developing  and  implementing  the  planning 
function  for  budget  considerations.   The  Office  of  Program 
Coordination  has  the  important  role  in  planning  at  the 
program  level.   At  the  Council  level,  it  is  verbalized  that 
planning  is  the  responsibility  of  all  --  the  Council,  panels, 
program  directors.   With  this  in  mind,  in  order  for  a  network 
of  meaningful  information  to  flow,  input  on  how  progress  is 
to  be  measured  must  be  provided.   Unless  evaluation  considera- 
tions are  taken  into  account  at  all  stages  in  planning  by 
these  groups,  two  results  can  be  expected.   First,  planning 
will  become  a  narrow  channel  providing  a  script  for  budget 
planners.   Second,  evaluation  studies  will  continue  to  be 
descriptive  studies  without  a  purpose  or  program  application. 
Staff  specializing  in  evaluation  approaches  should  be  involved. 
How  the  competence  should  be  developed  or  where  personnel  is 
stationed  should  not  cause  administrative  irritation.   Certainly 
the  Research  Division  must  develop  technical  competence  in 
evaluation  methods. 

Second,  postponing  the  identification  of  measures  and  indi- 
cators showing  objective  and  goal  attainement  tends  to  minimize  the 
role  of  evaluation  in  the  planning/management  evaluation  scheme. 
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Improved  planning  capability  and  increased  competence  in 
research  and  evaluation  are  evident  under  the  Office  of  the 
Deputy  Chairman  for  Policy  and  Planning.   Equally  visible  is 
an  increased  interest  in  planning  among  program  leaders  and 
the  Office  of  Program  Coordination  under  the  Deputy  Chairman 
for  Programs.   These  developments  lead  to  the  expectation 
that  the  role  of  evaluation  will  receive  more  prominence 
than  it  previously  has  attracted  —  not  less  prominence. 

Third,  the  substance  of  evaluation  programs  is  dependent 
upon  a  good  network  that  identifies  issues.   Top  management, 
the  Council,  the  panels,  and  the  program  directors  could  be 
more  effective  in  seeking  evaluation  capabilities  that  are 
problem  solving  oriented  at  the  policy  level.   Research  and 
evaluation  studies  should  be  expected  to  provide  assistance 
in  understanding  the  work  of  the  Endowment.   To  date, 
expectations  seem  especially  low  as  far  as  evaluation  studies 
are  concerned.   For  a  variety  of  reasons,  the  Research 
Division  has  been  relegated  to  being  an  entity  of  its  own. 
The  role  of  evaluation  should  not  be  considered  as  a  mere 
add-on.   The  research  and  evaluation  programs  should  be 
in  the  mainstream  and  the  function  oriented  toward  issues 
and  problems  in  goal  attainment  for  the  Endowment. 

Fourth,  the  Endowment  may  not  have  identified  and 
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communicated  its  data  requirements.   There  are  information 
sources  within  the  programs  and  at  the  general  administrative 
level.   Some  program  directors  feel  that  too  many  data 
collections  are  incomplete,  contain  inaccuracies,  and  are  not 
current.   Others  believe  their  needs  are  met  by  supporting 
non-agency  data  gathering  groups.   Some  program  leaders 
develop  their  own  capabilities  in  data  collection.   Internally, 
there  are  data  processing  developments  to  capitalize  on 
application  information.   And,  of  course,  there  are  the  usual 
fiscal  and  budget  and  grant  administration  data  collection 
operations.   There  are  contracts  for  developing  the  format 
for  gathering  data,  e.g. ,  the  National  Information  Systems 
Project.   Doubtless,  there  are  other  programs  underway.   Any 
program,  especially  research  and  evaluation  programs,  is 
dependent  upon  reliable  statistical  information.   That,  however, 
is  not  the  most  important  consideration.   The  basic  question 
is  to  determine  what  information  is  required  to  meet  the 
needs  of  the  Endowment. 

In  19  77  the  Research  Division  accepted  a  very  good 
final  report  on  a  feasibility  study  for  an  economic  data 
program  on  the  condition  of  the  arts  and  cultural  organizations. 
Recommendations  from  this  report  have  been  followed  in  research 
studies.   But,  it  does  not  appear  that  the  report  was  used  to 
define  data  requirements  for  the  Endowment.  To  clarify  and 
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integrate  the  thinking  about  information  requirements, ' it  is 
suggested  that  a  working  group  be  formed  under  the  direction 
of  the  Deputy  Chairman/Policy  and  Planning  with  representation 
from  all  data  users.   The  Research  Division  could  provide 
the  coordination  and  assist  in  identifying  linkages  among  the 
many  data  requirements.   It  is  necessary  to  emphasize  that 
evaluation  supportive  of  Endowment  policies  will  be  suspect 
until  these  data  acquisition  and  data  use  areas  are  better 
understood. 

Fifth,  in  implementing  evaluation  studies  in  a  systematic 
fashion,  there  should  be  less  separation  between  research 
and  evaluation  studies.   For  example,  a  program  of  research 
can  be  responsive  to  program  needs  as  well  as  emphasizing  the 
impact  analysis  requirements  for  Endowment  policy  planning 
and  decision  making.   Also,  the  designs  for  evaluation 
studies  can  be  of  high  quality  in  terms  of  technical 
considerations  and  problem  solution. 

The  term  evaluation  is  disturbing  to  many  managers  and 
leaders  for  reasons  stated  so  often. that  they  are  not  repeated 
here.   Emphasis  should  be  placed  on  analysis  of  data  and 
interpretation  of  information.   The  requirement  is  that 
researchers  and  evaluation  specialists  work  with  leaders  in 
defining  issues  and  purposes  that  require  solution.   The 
results  need  to  be  important  to  members  with  responsibility 
for  policy  issues  as  well  as  to  program  leaders. 
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Sixth,  program  leaders  conduct  self -evaluation  and 
internal  evaluation  studies  differently  with  varying  degrees 
of  quality  standards.   Some  programs  will  have  a  stated 
impact  requirement,  for  example,  Office  of  Partnerships, 
State  Programs.   Some  programs  may  not  have  explicit  evalua- 
tion statements.   Nevertheless,  self-evaluation  and  internal 
program  analysis  studies  at  the  program  level  should  be 
encouraged.   This  approach  tends  to  be  less  threatening  and 
is  useful  in  improving  program  effectiveness. 

Seventh,  a  plan  for  intensive  program  review  should  be 
implemented  to  supplement  the  continuing  data  collection  and 
evaluation  efforts  of  each  program. 

Such  a  plan  would  provide  an  in-depth  review  of  a  few 
programs  each  year  based  on  available  resources.   Innovative 
arrangements  could  be  devised  using,  for  example,  cross 
cutting  themes  such  as  fellowships,  touring,  etc.   A  schedule 
indicating  a  review  every  3  years,  for  example,  would  encourage 
program  staff  to  be  aware  of  and  prepare  for  such  a  review. 
In  this  way  the  network  communicating  results  on  issues 
enhances  chances  for  timely  use  of  such  results  in  policy 
discussions . 

Eighth,  provision  should  be  made  for  discrete,  once 
only  evaluation  studies.   This  is  necessary  to  meet  crises. 
It  is  unrealistic  to  believe  such  needs  will  not  occur.   Even 
with  a  first-rate  systematic  evaluation  approach  "crises" 
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responses  will  be  requested.   However,  a  research  and  evalua- 
tion unit  should  not  be  forced  to  expend  many  resources  on 
"one-shot"  studies. 

What  does  the  above  say  about  the  development  of  a 
systematic  program  in  evaluation? 

1.  Blend  the  research  program  and  the  evaluation  requirements 
so  that  the  purpose  of  these  activities  can  be  communicated 
in  terms  of  assistance  in  solving  Endowment  policy  issues 
and  problems. 

2.  Data  requirements  for  the  Endowment  should  be  clarified 
and  defined.   The  Research  Division,  as  a  member  of  an 
Endowment  wide  working  group,  should  provide  a  coordinating 
role . 

3.  In  support  of  all  groups  with  responsibilities  in  planning 
at  the  program  level,  develop  a  capability  for  technical 
assistance  in  the  phase  of  planning  dealing  with  objectives 
and  identifying  measures  of  objective  attainment.   This 
would  be  a  complementary  role  that  would  contribute  to 
improved  working  relations . 

4.  Seek  the  opportunity  to  provide  technical  and  professional 
assistance  to  groups  responsible  for  short  and  long  range 
planning  from  the  standpoint  of  laying  the  groundwork  for 
impact  studies  to  be  conducted  at  later  dates.   In  this 
way  bridging  the  gap  between  meeting  program  objectives 
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and  Endowment  goals  may  have  a  better  opportunity  for 
completion . 

5.  Develop  a  systematic  multi-year  program  of  evaluation 
research  that  is  directed  toward  Endowment  policy  and 
decision  making. 

6.  Develop  a  scheme  for  periodic  intensive  external  evalua- 
tion of  each  program  on  a  set  schedule   For  example, 
resources  might  allow  intensive  study  of  five  programs 
per  year,  to  be  repeated  every  three  years.   Innovative 
arrangements  could  be  devised  using  cross  cutting  themes, 
e.g.,  fellowships,  touring,  etc. 

7.  Reserve  resources  for  a  few  special,  one-of-a-kind  evalua- 
tions each  year  based  on  important  pressure  requirements. 
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Section  2 


A  requirement  in  the  work  order  for  consulting  services 
is  to  review  completed  as  well  as  on-going  evaluation  studies 
to  assess  procedures  and  circumstances  affecting  successful 
completion  of  the  projects.   These  studies  have  been  reviewed 
and  reactions  have  been  communicated  in  discussions  with 
members  of  the  Research  Division  staff.   Also,  discussions 
about  the  uses  of  the  results  of  evaluation  studies  have  been 
held  with  many  program  directors,  other  NEA  officials  and 
several  individuals  outside  NEA  who  are  knowledgeable  about 
arts  administration. 

The  negative  aspects  noted  in  reviewing  completed  evalua- 
tion studies  can  be  listed,  but  that  approach  has  not  been 
productive  in  this  situation  simply  because  the  conditions 
are  different  now  than  previously,  and  the  capabilities 
within  the  Research  Division  and  throughout  the  Endowment  are 
quite  improved  in  relation  to  the  period  that  the  Evaluation 
Unit  functioned  as  a  separate  entity.   It  would  be  interesting 
to  search  out  usable  evaluation  information  and  impact  find- 
ings and  trace  their  uses.   Tracing  uses  is  an  interesting 
technique  but  is  very  time  consuming  and  was  not  pursued. 
Informal  discussions  on  uses  of  findings  of  completed 
evaluation  studies  were  not  productive  either. 
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The  Evaluation  Report 

1.  The  final  reports  are  too  long  and  difficult  to  understand. 
This  statement  is  a  common  one,  and,  expected.   The 
reasons  for  the  lengthy  presentations  in  reports  are  many. 
Sometimes  the  sponsor  desires  a  wide  distribution  of  a 
report  and  often  encourages  long  programmatic  descriptions. 
Evaluation  reports  should  be  directed  toward  a  stated 
user  with  well  defined  purposes  and  the  contractor  should 
not  be  in  doubt  about  this. 

2.  Evaluation  reports  too  often  describe  activities.   The 
analysis,  assessment  and  interpretation  become  secondary. 
Program  staff  need  good  descriptions  of  project  activities 
to  meet  a  variety  of  requirements.   But,  evaluation 
studies  may  not  be  the  best  source  of  description  of 
program  activities.   Too,  there  may  be  confusion  in 
terminology  in  some  situations.   Simply  describing  all 
activity  is  not  assessing  its  effectiveness,  its  quality, 

or  its  impact.   There  are  two  considerations  here.   (A)  More 
precise  language  should  be  written  in  the  solicitations. 
Questions  listed  in  solicitations  too  often  encourage 
description.   (B)  The  language  of  evaluation  is  not  common, 
not  standard  among  the  program  staff  and  the  researchers. 
The  Research  Division  should  provide  a  common  vocabulary 
as  a  means  of  improving  internal  communication.   This 
seems  like  an  opportune  time  to  establish  a  common  technical 
language  in  research  and  evaluation. 
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3.   The  format  for  evaluation  reports  may  be  confusing  to 
a  contractor  and  may  be  dulling  the  interest  of  a 
targeted  reader.   For  example,  in  several  reports  the 
executive  summary  is  just  another  report  and  the  importance 
of  outstanding  impacts  becomes  less  distinguishable.   It 
would  be  better  to  direct  the  contractor  to  deal  with 
significant  findings.   The  Research  Division  should 
assume  responsibility  for  preparing  the  action  document 
when  the  report  is  forwarded  to  a  target  administrator. 
In  this  way  the  issues  can  be  restated,  the  policy 
problem  that  had  been  the  basis  for  the  evaluation  study 
can  be  laid  out  and  the  action  that  should  be  taken  can 
be  stated  by  the  competent  professional  within  NEA.   All 
of  the  above  should  be  performed  within  the  time  frame 
previously  established  by  the  administrator  responsible 
for  action  on  the  issue. 

The  Solicitation 

These  paragraphs  are  directed  toward  improvement  in 

solicitations  and  do  not  enumerate  the  many  good  elements  in 

the  format  in  use. 

1.   The  most  important  improvement  should  occur  prior  to 
preparing  and  issuing  the  solicitation.   The  policy 
issues  that  suggest  an  evaluation  study  should  be 
documented.   The  present  solicitations  state  questions 
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to  be  answered  but  the  statement  of  issues  and  the 
problems  to  be  solved  may  not  be  evident.   What  policy 
issue  is  being  discussed? 

The  name  of  individual  or  group,  e.g.,  a  Congressman 
wanting  an  evaluation  of  a  program  may  be  known  to  the 
staff  but  this  may  not  be  documented.   Statements  on 
issues,  a  statement  of  problems  to  be  solved,  who  wants 
the  evaluation,  and  what  action  is  required  are  items 
to  be  accepted  and  approved.   Agreement  on  these  topics 
would  help  to  understand  the  needs  of  top  management, 
i.e.,  is  the  evaluation  study  an  expedience  dealing  with 
funding  or  is  it  intended  to  clarify  the  role  of  the 
Endowment? 
2.   The  substantive  content  of  the  solicitation  would  profit 
from  more  critical  analysis.   The  suggested  questions  to 
be  answered  in  a  solicitation  should  not  be  a  long 
shopping  list.   The  staff  should  discriminate  between 
information  required  and  why  and  information  that  would 
be  "nice  to  know. " 

The  Design  of  Evaluation  Studies 

There  are  few  comments  concerning  the  design  of  completed 
and  on-going  evaluation  studies.   The  methods  of  sampling  and 
the  approaches  used  vary  in  quality,  as  expected.   As  the  NEA 
staff  gain  more  experience  and  acquire  more  technical  competency 
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in  evaluation  methods,  data  analyses  and  interpretation  will 
improve.   Even  so,  the  turning  point  in  using  the  results  of 
evaluation  studies  will  be  the  astuteness  in  the  design  for 
evaluation  studies.   There  is  no  reason  to  believe  that 
questionnaire  surveys  will  be  discontinued  but  there  will 
not  be  many  questions  that  ask  a  respondent  simply  whether  or 
not  the  grant  funds  received  were  useful.   Also,  for  many 
years  NEA  will  depend  upon  secondary  sources  for  data  for  use 
in  analysis  but  the  responsible  NEA  staff  will  have  a 
better  understanding  of  the  accuracy  and  comprehensiveness  of 
the  data  collections  available.   Too,  the  quality  of  evalua- 
tion proposals  should  improve  as  there  are  several  research 
groups  around  the  nation  with  good  track  records  in  evaluation 
research  and  eager  for  business.   Be  more  agressive  in  selection 

Monitoring  a  Contractor 

The  important  work  of  the  staff  is  done  up  front  —  the 
pre-solicitation  definitions,  the  staff  decisions  on  acceptable 
possible  methodologies,  the  clarity  and  tightness  of  the 
solicitation,  and  the  definiteness  of  design  during  contract 
negotiations.   Assistance  to  the  contractor  in  implementing 
the  study  continues,  of  course,  but  after  the  up-front  work 
is  completed,  it  is  better  to  stay  out  of  the  way,  to  keep 
informed  and  helpful  so  that  the  need  for  intervention  is 
unlikely.   The  research  staff  is  competent  and  the  need  for 
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further  discussion  is  unnecessary.   However,  because  evaluation 
results  are  feared  by  some  and  there  are  those  who  feel  that 
evaluation  studies  have  little  value,  a  review  of  the  philosophy 
and  strategies  in  monitoring  is  suggested.   For  example,  it 
isn't  useful  to  complain  about  the  use  of  a  data  source  after 
the  study  is  completed.   It  doesn't  do  much  good  to  ask  a 
contractor  for  monthly  statements  of  progress  if  the  material 
is  not  used  except  for  administrative  protection. 

Finally,  the  evaluation  studies  to  date  may  have  been 
used  in  more  ways  than  have  been  suggested.   For  example, 
information  may  be  used  for  general  purposes  in  preparing 
program  reports.   The  fault  is  that  the  results  of  evalua- 
tion studies  have  had  limited  acceptance  by  the  Endowment 
staff  and  have  been  given  a  place  of  low  value  as  aids  .in 
solving  problems.   Now,  it  can  be  hoped  the  thrust  will  be 
different. 


